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ABSTRACT 





The Military Traffic Management Command (MTMC) is currently 
reengineering the DOD Personal Property Shipment and Storage Program. DOD 
is conducting three pilot projects in support of this effort. Each pilot project 
represents a policy alternative for improving moving services for military families. 
The Deputy Under Secretary of Defense (Logistics) tasked United States 
Transportation Command (USTRANSCOM) to evaluate the personal property 
pilot programs as part of Management Reform Memorandum # 6. This thesis 
evaluates the policy alternatives for reengineering the DOD personal property 
program using a stakeholder approach. 

The study develops a model and establishes criteria for evaluating the three 
policy eaaalices Values are determined for the criteria by interviewing a 
sample of stakeholders from business, government, and customers. Policy 
alternatives are then analyzed from each stakeholder perspective. Finally, a policy 
alternative is identified that best satisfies the criteria for aaah stakeholder as well 


as the aggregate of stakeholders. 
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I. INTRODUCTION 


A. PERSONAL PROPERTY SHIPMENT AND STORAGE PROGRAM 

The Department of Defense (DoD) Personal Property Shipment and Storage 
Program is a $1 billion dollar program managed for the military services by the Military 
Traffic Management Command (MTMC). DoD is the moving and storage industry’s 
single largest eineias with over 650,000 shipments a year -- approximately 15 
percent of the moving industry business (Moore, 1995). The servicemember’s quality 
of life is effected over 650,000 times a year by the policies that govern the movement 
and storage of their possessions. Over a military career, a servicemember is typically 
required to move more than 20 times (Moore, 1995). 

1. Background of Reengineering Effort 

The DoD’s Quality of Life Task Force, established by Secretary of Defense 
William J. Perry and under the leadership of Mr. John O. Marsh, a former secretary of 
the Army, mandated that systems be simplified, the quality of service for the 
servicemember be improved and, most of all, that members be satisfied customers of 
DoD’s services. The National Performance Review and the Acquisition Reform 
Initiative have also recognized the need for DoD to improve the processes used to 
regulate and manage commercial programs. 

Secretary of Defense (SECDEF) William S. Cohen, in his Defense Reform 
Initiative Report, November 1997, stated that the Department must do a better job of 


moving DoD servicemembers and their families to their new duty stations. The report 
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states in part, “...despite the fact that DoD moves more household effects than any 
United States corporation, the system we have created to do the moving has given our 
personnel some of the worst service in the nation (Cohen, 1997).” Improving the 
current personal property program remains a critical quality of life issue, and is listed as 
one of the SECDEF’s nine Defense Reform Initiatives. 

Under the current program, many shipments are delayed, lost, stranded, burned, 
and damaged (Moore, 1995). Servicemembers and their families consistently 
experience poor service from movers, excessive incidence of loss or damage to their 
property, and a laborious claims process (GAO/NSIAD-97-49, 1996, p. 1). The failures 
of the current program have caused hardships throughout the military for 
servicemembers and their families. 

MTMC is currently reengineering the program to improve the quality of these 
moves and reduce costs, while attempting to not adversely impact industry. Efforts to 
reengineer the personal property program have involved numerous governmental and 
industry organizations including Congress, the Secretary of Defense, the Military 
Services, industry trade associations, and individual transportation service providers. 
Though MTMC has oversight of the reengineering pilots under the cognizance of the 


United States Transportation Command (USTRANSCOM), MTMC, the Army and the 


Navy are each responsible for conducting separate pilots to evaluate policy alternatives. 

















2. Organization of the Personal Property Program 

MTMC is ; jointly staffed, major Army command under USTRANSCOM. 
While MTMC is responsible for the overall operation of the Personal Property 
Shipment and Storage Program, each Service runs offices that provide customer support 
to servicemembers in the process of moving. MTMC serves as the single contracting 
authority for personal property. 

Although the DoD’s domestic personal property program functions under 
centralized MTMC management, operations are carried out at 153 Military Service- 
managed Personal Property Shipping Offices (PPSOs) (Beyer and Schwartz, 1993, p. 1- 
2).' Approximately 1,100 commercial carriers move the household goods of military 
members throughout the United States each year. 

Every six months, MTMC solicits bids from carriers for specific domestic traffic 
lanes (routes). In response, MTMC receives more than 1.1 million bids for domestic 
movements from commercial carriers. Rates, based on poundage, mileage, packing, 
labor, and storage, are established and proffered to moving and storage companies by a 
MTMC contracting officer. 

A successful bid to MITMC’s solicitation places the contractor on a local area 
listing. A rotational system is used by PPSOs to equitably distribute business among 


accepted carriers. Every domestic PPSO maintains a Traffic Distribution Record (TDR) 


' The 153 PPSOs in the United States consist of 45 Army, 30 Navy, 609 Air Forces, and 9 Marine Corps. The 
Coast Guard also operates 21 PPSOs. 
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on which it records shipment awards. The TDR lists carriers according to their filed 
rates, with the lowest ‘aie carriers listed first. 

The moving industry contains van lines with agent networks, dependent 
carriers, agents, and forwarders. Six major carriers dominate the moving industry. In 
1996, these carriers accounted for 83 percent of the market. Many small businesses in 
the moving industry are agents for one of the major carriers. Many carriers create 
“paper companies” to bid and gain a higher percentage of the traffic allocation while 


the parent company conducts the move. 


B. DESCRIPTION OF PILOT PROJECTS 

Since 1994, DoD has been engaged in initiatives to reengineer the personal | 
property program. On June 21, 1994, the Deputy Commander-in-Chief, 
USTRANSCOM, directed the Army component of USTRANSCOM, the Military 
Traffic Management Command and program manager for DoD’s Personal Property 
Shipment and Storage Program, to reengineer the personal property program 
(GAO/NSAIAD-97-49, p. 30). MTMC’s initial efforts met with strong industry 
resistance and heightened congressional interest. One June 15, 1995, the House 
Committee on National Security, concurred that DoD must pursue a higher level of 
service, and directed that DoD iadeake a pilot program to implement commercial 
business practices and standards of service. 

Based on congressional language in the Fiscal Year 1996 National Defense 


Authorization Act, DoD eventually developed three pilot programs to reengineer the 
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Household Goods movement process (Personal Property Program Management Reform 
Memorandum #6, 1999). These pilots are 1) the Military Traffic Manasement 
Command pilot; 2) the Navy’s Servicemember Arranged Moves (SAM) pilot; and 3) 
the Army’s Hunter Airfield pilot which was later expanded into the Full Service 
Moving Project (FSMP). Figure 1 provides an organizational overview of the pilot 


project hierarchy. 





Office of the Secretary of Defense 
Deputy Under Secretary of Defense 
(Logistics) 


Office of the Secretary of Defens 
Transportation Policy 





USTRANSCOM 
MTMC 


SAM Pilot Hunter Pilot ; 
(Full Service Move Project) 


Figure 1. Pilot Project Hierarchy 





1. MTMC Reengineering Effort 
The MTMC project is attempting to reengineer the current program in order to 
improve the move process. MTMC’s pilot aims to significantly improve the quality 


assurance and procurement methods of the personal property program through the use 
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of a Federal Acquisition Regulation (FAR)-based “best value” contract. The key 
features of this ene include selecting carriers based on servicemember satisfaction 
and past performance rather than simply price; achieving stronger carrier commitment 
with long-term contracts; and offering full replacement value protection and direct 
claims settlement. 

MTMC’s final contract solicitation for this pilot was released 14 March 1997. 
Although MIMC received proposals in June 1997, pilot implementation was delayed 
due to strong industry resistance and Congressional concerns. Concerned about the 
impact that the competition system and any nonstandard commercial business practice 
requirements might have on small businesses, the statement of managers accompanying 
the 1997.DoD Authorization Act directed that DoD report on the impact of the pilot 
program on small business (Warren, 1999). After reviewing the defense reports on 
waa business impact, the House Committee on National Security was still concerned 
that MTMC’s pilot program did not satisfactorily address issues raised by small moving 
companies. As a result, Congress directed that the Secretary of Defense establish a 
working group of military and industry representatives to develop an alternative pilot 
project (Warren, 1999). 

Although the working group reached a consensus on many issues, including a set 
of program goals, it could not reach agreement on the approach to take for the pilot test 


(Warren, 1999). Consequently, the two sides presented separate proposals. In 


November 1996, the General Accounting Office (GAO) reported that MTMC’s 








proposal met the goals of reengineering the personal property program to a greater 





extent than the industry plan (GAO/NSIAD-97-49, p. 4). 
The MTMC pilot program implementation, delayed by numerous bid protests, 


finally commenced in January 1999. The pilot reengineers the existing program for 50 





percent of the moves originating in three states: North Carolina, South Carolina, and 
Florida. The pilot runs concurrently with the existing MTMC-managed program at the 
installations in the three states and will involve approximately 18,500 annual moves 


(Warren, 1999). The pilot is scheduled to run for one-year with two one-year option 





periods. (BGen. Privratsky Information Paper, 1999). 

2s Sailor Arrangement Move (SAM) 

The Department of Navy initiated a separate pilot effort in 1997 to test the 
option of allowing servicemembers to choose their own carrier as well as coordinate 


their move with that carrier. The Servicemember Arranged Move (SAM) pilot is a low 





volume pilot based on voluntary participation featuring payment for services by an 
international merchant’s procurement authorization card IMPAC). The servicemember 
selects a mover from a list of carriers that “ self-certified as small businesses and 
approved by MTMC. Servicemembers are then able to coordinate the move directly 
with the mover. 

Commencing in January 1998, the option is offered for shipments originating in 
the areas surrounding Puget Sound, Washington; San Diego, California; Norfolk, 
Virginia; and Groton, Connecticut (GAO/T-NSIAD-99-106, 1999, p. 4). Eligible 


participants must limit shipments to at least 3,000 pounds and to costs between $2,500 


and $25,000 (Warren, 1999). Furthermore, this pilot excludes shipments from non- 
temporary storage or warehouses. 

The pilot is not intended to replace the existing MIMC-managed pro —_— and is 
limited to Navy military members with permanent change of station orders from the 
participating site. Attempts to expand the program to include all servicemembers and 
DoD civilians, and to incorporate additional objectives were not approved. 

3. Full Service Move Program (FSMP) 

Separate from the MTMC pilot program, the Department of the Army is 
attempting to determine whether the commercial business practice of outsourcing can 
alleviate known problem areas. The pilot was initiated in February 1996 as a Quality of 
Life initiative to improve the relocation process and to test commercial business 
practices in a military environment (Warren, 1999). The pilot effects a total of 1,400 
annual moves that originate at the Hunter Army Airfield, Savannah, Georgia. 

On January 31, 1997, the Army selected PHH Relocation (now called Cendant 
Mobility) as the contractor for the pilot. Cendant Mobility is the largest 
relocation/move management company in the world. In July 1997, Cendant Mobility 
began providing relocation services at Hunter Army Airfield. These services consisted 
of point-to-point move management, personal move counseling and coordination, 24 — 
hour in-transit visibility of the shipment, direct claims settlement with the service 
member, full replacement value insurance, and a single point of contact for the 


servicemember (Warren, 1999). Additional relocation management services were 


optional to soldiers and included services such as home finding for buyers and renters, 














mortgage services, marketing services for home selling, and property management. 
Throughout the entire relocation process, the contractor’s move counselor is the 
soldier’s single point of contact, from origin to destination, from entitlements 
counseling to claims settlement (LGen. Coburn, 1999). 

The initial Hunter Test was for one year, ending 30 June 1998. At the end of the 
initial test, an extension option was exercised for another year. On February 12, 1999, 
DoD stated that it intended to expand the Hunter Test into the Full Service Moving 
Project (FSMP) with certain modifications based on earlier lessons learned from the 
Hunter Test. Moreover, the Assistant Deputy Under Secretary of Defense, 
Transportation Policy took over sponsorship of the pilot from the Army. When fully 
implemented, FSMP will involve all military Services as well as the Coast Guard. Test 
areas in consideration are Georgia, North Dakota, and the utaal Capital Region, 
which include approximately 45,000 annual moves. The project will allow DoD to test 
and evaluate a larger move volume as well as Service-unique requirements under the 
commercial business practice concept (FSMP Overview, 1999). The FSMP is expected 
to commence in late 1999. 

4. DoD Pilot Project Evaluation Process 

The USTRANSCOM is tasked by the Deputy Under Secretary of Defense 
(Logistics) with overseeing all personal property pilots, ensuring consistency in 
evaluation criteria and assessment, evaluating the pilots to determine which pilots or 
tests, or portions thereof, could provide better long-term results, and then 


recommending the follow-on course of action and time line for implementation 


throughout DoD (Evaluation Plan for Reengineered DoD Personal Property Pilot, 
SAM, and DoD FSMP, 1999). Presently, USTRANSCOM is working with the services 
to develop a common set of data measures to evaluate the pilot project's results 
(GAO/T-NSIAD-99-106, p. 5). | 

Evaluation methodology currently concentrates on four main areas: quality of 
life, costs, impacts on small business, and process improvement innovations (LGen. 
Thompson, 1999). Quality of life is the most heavily weighted of the four factors and 
will be measured by the servicemember’s direct feedback as to whether he/she would 
use that carrier again (LGen. Thompson, 1999). Total cost includes direct and indirect 
costs. Direct costs include contracted services and goods such as transportation, storage 
and accessorial services. The indirect baseline cost includes management of the 
personal property program, quality assurance, financial transaction processing, and 
additional systems operations and maintenance costs. The third factor considered, the 
impact on small business, will be measured against the Small Business Act, Public Law 
855536 that establishes a government-wide goal for small business participation at not 
less ‘ian 23 percent revenue (LGen. Thompson, 1999). The USTRANSCOM 
evaluation program will also identify unique features of each pilot program that 
substantially improves the service to the customer. These features include, but are not 
limited to: the use of move management services, electronic data interchange capability 


and payment features. 
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Although standard data will be collected from each pilot, the specific methods of 





data collection may differ somewhat from program to program (LGen. Thompson, 
1999). Furthermore, data will be collected and compiled for each pilot by a different 
organization. The initial Hunter pilot used Electronic Data Systems (EDS) to evaluate 
servicemember quality of life, cost, and small business participation as compared to the 
current household goods move process. The FMSP will use the installation 


transportation office / traffic management office (TMO) and the relocation service 





contractor to collect the baseline and test data. This data will be compiled and validated 
by the Army Audit Agency. The SAM program will have the naval base personal 
property shipment offices compile the baseline and test data. This data will be 
forwarded to and validated by the Naval Audit Service. GAO has repeatedly 


recommended that the Secretary of Defense develop a comprehensive strategy to test 





each of the policy alternatives (GAO/NSIAD-99-138, 1999, p. 8). 

Ultimately, all test data will be forwarded to USTRANSOM for evaluation. A 
USTRANSCOM contractor will design and conduct surveys, develop and implement 
the evaluation plan, and prepare an integrated recommendation plan for a new DoD 
personal property program (Evaluation Support for DoD’s Personal Property 
Reengineering Programs Scion of Work, 1999). An integrated product process 
team (IPPT), composed the O-5/O-6 level representatives from the Office of the 


Secretary of Defense, USTRANSCOM, the Services, DoD Inspector General, and 


I] 


Audit Agencies will analyze the comparative test data, develop options, and forward 
them to an Executive Working Group (EWG). | 

The EWG will consist of Flag/General Officer level representatives from 
USTRANSCOM, the Services and OSD (LGen. Thompson, 1999). The charter of the 
EWG is to make final recommendations for a el DoD personal property program to 
Commander-in-Chief (CINC), USTRANSCOM (LGen. Thompson, 1999). 

CINC, USTRANSCOM will approve or make modifications to EWG’s 
recommendation(s) after a review. As the DoD Executive Agent for the evaluation of 
all DoD personal property pilot programs, CINC, USTRANSCOM will submit a final 
recommendation(s) to the Secretary of Defense by December 2000. 

5. Summary 

Since 1994, DoD has pursued initiatives to reengineer the personal property 
program. DoD has attempted to involve industry in these efforts, but there has been 
minimal agreement on pilot characteristics. DoD is currently testing three different 
approaches to reengineering the personal property program under separate pilot 
projects. Figure 2 summarizes the characteristics of these pilots. These projects effect 
approximately 21,000 annual moves. The expansion of the Hunter pilot into the FSMP 
will increase participation in the pilots to approximately 66,000 annual moves or 10 
percent of total annual moves.” Final recommendations will be forwarded to the 


Secretary of Defense from CINC, USTRANSCOM by December 2000. 


? The personal property program covers approximately 650,000 annual moves. 


412 








SAM 


FAR-based contract- 
commercial LOA 

Full Value Protection 
(Maximum $72K) 

Direct claims settlement w/ 
carrier 

Claim settlement w/in 60 
Carriers approved and in good 
standing with HOMTMC 
Inclusion of the Service 
Contract Act 

Inconvenience payments 
Customer satisfaction surveys 
Tailored pre-move counseling 
by PPSO 

Shipment tracking (ITV) 
establishment of a toll free # to 
be manned five days a week 
(M-F) from 0700 to 1800 
Central Time 

Use of Pagers to support direct 
delivery 

Pricing strategy based on 
commercial tariff 

Weight restrictions reduced 
from 3000 to 1000 Ibs. For 
(HHG) 

SIT increased from 90 days to 
360 days 

Servicemember chooses carrier 
from among available 
participating carriers 
Servicemember may interview 
and/or visit carrier 

Carriers may service any or all 
channels 

100% small business 
Government Credit Card 
payment 


Figure 2. Program Comparison 


Hunter (FSMP 


FAR-based contract 

Full Replacement Protection 

(Maximum $75K) 

Direct claims settlement w/ move 

manager 

Claims settlement w/in 30 days 

The Contractor obtains est. for 

determining the replacement 

value/repair cost of the property 

Transportation provider qualifications: 
Financial stability 

e  Ensured capacity 

e Past performance standards 

e Loss/damage prevention 
measures 

e Guaranteed on-time pick-up and 
delivery 

e Inconvenience payments 

Move Manager qualifications: 

e Performance risk assessment 

e Past performance 

e Ability to meet SOW 
requirements 

Monthly Move Analysis Report 

Pre-move counseling 

Point to point move management 

Binding Estimates for excess costs 

Destination information 

Shipment tracking via a toll-free # and 

an electronic address. Service shall be 

available 24 hours a day, 7 days a 

week 

Real Estate Services 

Domestic pricing based on 

commercial tariff 

International pricing based on Single 

Factor Rate 

Transportation providers pay 

commissions to move managers 

Distribution based on Quality 

Use of move management service 

1 year agreement with transportation 

providers 

2 year contract w/ 3 option years with 

move manager 


Contractor required to meet small] 


business goals in accordance with SBA 
Use Bank Service to pay 
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MTMC 


FAR-based contract 

Full Replacement Protection (Maximum 

$63K) 

Claims settlement w/ Government 

The Contractor obtains est. for 

determining the replacement 

value/repair cost of the property 

Carrier qualifications 

® Committed Daily Capacity 

e Past performance standards 

e Established quality measures 

e Personnel requirements 

e Carriers evaluated on on-time 
pickup & delivery and damage 
rates 

° Inconvenience payments 

Use of Customer Survey 

Quarterly performance reviews 

Claims Activity Report 

Pre-move counseling 

Shipment tracking (ITV) toll free # that 

is operational 7 days a week, 24 hrs/day. 

Must response during working hours. 

Pricing Strategy based on commercial 

tariff 

Traffic allocated on a rotational 

capacity to the contract min. 

guaranties meet then on best value 

EDI: Transportation Operational 

Personal Property Standard System 

(PTOPS) 

Payment through DFAS 

Compliance with SBA 





C. PURPOSE OF THESIS 

The DoD is planning or conducting three pilot projects to reengineer its personal 
property shipping program. Each pilot project represents policy alternatives for 
improving moving services for military families. MTMC is responsible for evaluating 
all of the pilots; its intention is to determine which one could provide better long-term 


service (GAO/NIAD-99-138, pg. 2). 


D. RESEARCH QUESTIONS 

The primary research question of this thesis is: Which policy alternative for 
reengineering the DoD’s personal property program best satisfies stakeholder 
criteria? 


Subsidiary research questions supporting this issue are: 


1, What are the policy alternatives for reengineering the personal property 
 -program? | 
2; What are the criteria for evaluating for policy alternatives? 


3: Who are the stakeholders for the reengineering effort? 
4. What value do the stakeholders place on the evaluation criteria? 


5. Which policy alternative fulfills stakeholder criteria? 


E. SCOPE AND METHODOLOGY 

This thesis provides background on the DoD’s effort to reengineer the personal 
property program, identifies the dominant stakeholders of this effort, develops a single 
set of stakeholder criteria, and evaluates the alternate proposals with respect to these 


criteria. 
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This effort employs the basic tools of evaluation research. Specifically, the 





design involves collecting individual opinions and archival data using primary and 
secondary sources on three alternate proposals to reengineer the personal property 
program. The archival data include the study of government and industry documents. 
Using this documentation, a single set of criteria is developed to evaluate the policy 


alternatives. 





A stakeholder map is developed to identify dominant stakeholders. Individuals 
within each dominant stakeholder group are chosen using a convenience sample and 
interviewed using a telephonic questionnaire. The collected data are coded and 
introduced into a spreadsheet. Measures of central tendency are calculated for each 
criterion. Based si the value stakeholders’ place on the criteria, one proposal is 
identified that is the best fit for each stakeholder group’s expectations. Furthermore, 


one proposal is identified that is the best fit for all major stakeholders. 


F. LIMITATIONS 

Recommendations are limited to the current pilot projects. Stakeholder 
expectations are determined in terms of the pilot projects. There are additional 
stakeholder criteria but since they were not incorporated into the policy alternatives, 
they were not included in this study. This study focuses on determining which policy 
alternative(s) best reflect the stakeholders’ concerns and meets their needs. This study 


does not attempt to address problem definition nor implementation issues. 
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G. ORGANIZATION OF STUDY 

Chapter I, Introduction, presents the focus of this study. This chapter introduces 
the effort to reengineer DoD’s personal property program, describes the pilot projects © 
and outlines DoD’s policy evaluation process. Furthermore, this chapter establishes the 
research question and methodology, and presents an outline of the thesis. 

Chapter II, Literature Review, provides a definition and an overview of policy 
analysis in the public sector. An analysis process is also identified and reviewed. This 
chapter will provide an overview of the stakeholder concept. A method for conducting 
a stakeholder audit is described. _ 

Chapter III, Methodology, articulates the research methodology used to conduct 
this study. A goals and evaluation criteria map is presented. The extent to which each 
pilot project fulfills specified criteria is determined. Finally, a stakeholder analysis 1s 
conducted. 

Chapter IV, Data, presents summary statements and data garnered from the 
telephonic interviews. Measures of central tendency are developed for each stakeholder 


group based on the evaluation criteria. 


Chapter V, Analysis, discusses and analyzes the data collected in the telephonic 


interviews. 


Chapter VI, Conclusions and Recommendations, provides conclusions that can 
be drawn from an analysis of the data. Specifically, this chapter will recommend the 


policy alternative(s) that best fulfills stakeholder expectations. 
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Il. LITERATURE REVIEW 





A. INTRODUCTION 

Public policy analysis or evaluation determines which alternative public 
policies will optimize on a given set of goals (Nagel, 1990, p. ix). Related 
concepts include policy studies, program evaluation, public management science, 
and policy science. Policy evaluation and analysis emphasizes examining 
alternative public policies, as contrasted to describing them or explaining why they 
exist (Nagel, p. x). This chapter provides an overview of public policy analysis in 
the public sector, introduces a model for performing policy analysis, and 


demonstrates how this model can be applied to the research question. 





B. POLICY ANALYSIS IN THE PUBLIC SECTOR 

Public policy analysis is a systematic approach to making policy choices in 
the public sector (Walker and Fisher, 1994, p. 1). Its purpose is to assist 
policymakers in choosing a preferred course of action amid multiple complex 
alternatives. The policy analysis process gathers and presents the information that 
assists the policymaker in making a decision between policy alternatives. 

Much of the growing literature in the area of public policy addresses 
various approaches to policy analysis. Weimer and Vining identify three 
approaches: (1) the objective model, (2) the client advocate model, and (3) the 
issue advocate model (Weimer and Vining, 1991, p. 18). This is similar to the 


approaches described by Fischer, Frank and Forester: (1) the science model, (2) 
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the counsel model, and (3) the advocate model (Fischer, Frank, and Forester, 
1987, p. 138). | 
The objective or science model limits the analyst to questions that science 
can properly answer; that is, to questions of empirical fact and practical means 
(Fischer, Frank, and Forester, p. 137). The client advocate or counsel model 
focuses on policies that provide the greatest benefit to their client. Finally, many 
authors endorse the issue advocate model. This approach promotes policies that 
achieve the greatest good for society as a whole. In general, the analyst need ni 
adopt any one of the approaches in its extreme form. Rather io salle’ one of 
the three fundamental approaches, the analyst should attempt to keep all three 
under eaicicetien (Weimer and Vining, p. 19). 

There are several alternative ways to analyze policy options. They 
allcontain similarities, but each uses different key words to describe a particular 
framework and each goes into varying degrees of detail (Bonser, McGregor, and 
Oster, 1996, p. 48). The policy evaluation process generally involves performing 
the same set of logical steps, not always in the same order. Walker and Fisher 
present the following model as seen in Figure 3. This model is very similar to the 
process laid out by Stokey and Zeckhouser, Mason and Mitroff, and by Bonser, 


McGregor and Oster. 
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1. Identify the problem 
2. Identify the goals of the policy 
3. Decide on criteria | 


4. Select the alternative policies 


5. Analyze and compare each alternative 


6. Recommend the preferred alternative 


Figure 3. Generic Policy Evaluation Model 
















C. MODEL FOR EVALUATING POLICY ALTERNATIVES 

The framework shown in Figure 4 is a hybrid drawn from several 
approaches. The available literature provides the basic components of this model. 
However, this model places a different emphasis on some steps and integrates 


other steps to provide a rational and logical basis for evaluating this case. 
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1. State the problem 
2. Identify solutions 
3. Develop evaluation criteria 


4. Evaluate solutions based on criteria _ 





5. Recommend a preferred solution 


Figure 4. Proposed Policy Evaluation Model 


1. State the Problem 

The first step sets the boundaries and the context of the problem to be 
studied. It involves identifying the questions or issues involved, and the desired 
end state or goals. Proposals for policy changes generally arise out of the 
dissatisfaction of one or more groups of stakeholders with the efficiency or 
effectiveness of existing policies or programs (Rossi and Freeman, 1989, p. 73). 
Understanding the problem involves investigating the symptoms that prompt 
stakeholder dissatisfaction (Weimer and Vining, p. 212). An especially critical 
consideration in the explication of the problem is identifying the gap between the 


current policy or program outputs and outcomes and the desired output and 


outcomes. 








tC 


Program planning is predicated on a need to reduce the gap between the 
desired state of affairs and what actually exists--in other words between a goal and 
reality (Rossi and Freeman, p. 121). Goals are generally abstract idealized 
statements of outcomes (Rossi and Freeman, p. 121). Policy evaluation is often 

_ complicated by the existence of multiple goals that are not clearly defined. Many 


programs, especially those with a large target population or far-reaching effects, 





initially state their goals in broad and often vague terms. Stakeholders with 
varying and sometimes conflicting needs, interests, ind perspectives define the 
goals of a program. 

2. Identify Solutions 


This step identifies alternative policies that provide concrete steps to solve 





the stated problem. The majority of literature focuses on developing alternative 
solutions. These authors provide a sequence of activities that identify, design, and 
screen possible policy alternatives. However, as Weimer and Vining point out 
existing policy proposals often exist. “There are really four sources for 
developing policy alternatives: (1) existing policy proposals; (2) generic policy 
solutions; (3) “modified” generic policy solutions; and (4) custom-made solutions 

~ (Weimer and Vining, p. 225).” The policy analyst must be careful to avoid 
starting with a preconceived idea of the preferred alternatives even when 


evaluating existing policy proposals (Bonser, McGregor, and Oster, p. 49). 


Zi 


Moreover, the policy alternatives should be mutually exclusive. “You 
should ensure that your alternatives are mutually exclusive; they are, after all, 
alternative policies (Weimer and Vining, p. 227).” Alternatives are obviously not 
mutually exclusive if their focus or objectives are similar. In such circumstances, 
these policies may not provide distinct choices, and should be combined or 
eliminated. 

3. Develop Evaluation Criteria 

For a worthwhile evaluation to be undertaken, goal statements must be 
refined and stated in terms that can be measured, that is, operationally defined 
(Rossi and Freeman, p. 121). “For evaluation purposes, goal setting must lead to 
the operationalization of the desired outcome; that is the condition to be dealt stil 
must be specified in detail, together with one or more criteria (Freeman and Rossi, 
p. 121).” Criteria are required to evaluate the extent to which each alternative 
policy solves the stated problem. 

This step is especially difficult when diverse groups have agreed on broad 
goals but cannot agree on the objectives or sub-goals that support those goals. In 
this step the analyst must decide on the sctentia with which to evaluate alternative 
policies (Walker and Fisher, p. 2). “Loosely speaking, a policy is a set of actions 
taken to solve a problem (Walker and Fisher, p. 2).” The policymaker has certain 


criteria that, if met, would “solve” the problem. 
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Identifying these criteria is a critical step in evaluating competing policies. 
When multiple competing stakeholders are present, it is necessary to create a 
single set of measurement criteria by which alternative policies may be compared. 
Therefore, establishing criteria requires either assumptions or knowledge about 


two fundamental aspects of the situation: policy goals and stakeholder values. 





| Policy goals incorporate the set of actions that represent a solution to the problem. 
Stakeholder values provide a means to evaluate the policy alternatives Roni the 
perspective of each stakeholder. 
4. Evaluate Solutions Based on Criteria 
In this step, the alternative policies are analyzed and compared according to 


established criteria. “When the analysis of several alternatives has been 





completed, the policy analyst begins to synthesize the results (Bonser, McGregor, 
and Oster, p. 50).” Each alternative must be ranked according to the criteria 
chosen by the analyst in order to determine the alternative that is most likely to | 
fulfill the criteria (Bonser, McGregor, and Oster, p. 50). 

2. Recommend the Preferred Alternative 

Finally, the evaluator should recommend the preferred alternative. 
“Choosing the best alternative is trivial when you have an alternative that ranks 
highest on all criteria (Weimer and Vining, p. 234).” Unfortunately, this rarely 
occurs. Rarely is the best policy also the preferred policy. “Although you may 


sometimes be pleasantly surprised, you should expect to find different alternatives 
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doing best on different criteria (Weimer and Vining, p. 234).” When alternative 
policies fulfill the established criteria, but to differing degrees, the analyst must 
make a determination as to which criteria should be given greater weight. This 
determination is often subjective and based on the evaluator’s approach or 
perspective. In this situation there are implicit trade-offs between criteria. The 
evaluator must be explicit in stating the values and trade-offs that were made in 


determining the preferred alternative. 


D. APPLICATION OF MODEL TO STUDY 

The policy evaluation model developed in Figure 4 is used in this study to 
determine which policy alternative for reengineering DoD’s personal property 
program best satisfies stakeholder criteria. Figure 5 identifies the steps in the 
model and the supporting tasks. Figure 5 also provides an overview of each step 
in terms of the chapter location, the corresponding figures, the responsible party, 


and the procedure used. 
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» Methodology Overview | 


Method? 






















1. State the problem ChaptrI&0 | = | 
2. Identify solutions Chapter I i ase | 
3. Develop evaluation criteria | ChapterIIT | |= = |Researcher | | 

a. Identify goals 


b. Establish evaluation 
criteria 
c. Assess policy alternatives 


Forward Mapping Process 
Scale used to rate criteria for 
each pilot project 
Constructed stakeholder map 
Used single format for all 
interviews 


Convenience sample 


Telephonic interviews 
conducted; scale used to 
determine stakeholder values 
for criteria 


Collected data coded into 
spreadsheet 
Archival Research 


ee ee 

Data presented at goal & 
=. | sub-goal level 
Absolute value of differences 
between stakeholders’ and 
policy alternatives’ values is 


calculated 


Figures 8 & 
9 
Figure 10 


Figure 11 
Appendix B 


d. Conduct stakeholder 
analysis | 
(1) Identification o 
stakeholders 
(2) Determining 
stakeholder values 
(a) Develop a 
questionnaire 
(b) Choose a sample 
(c) Conduct 
interviews of 
sample 














(d) Classify results Appendix C 

(e) Validate results 
4. Evaluate solutions based on 
criteria 


a. Present Data Chapter IV 


b. Data Analysis Chapter V Figure 12, 
Appendix D 


Figure 5. Methodology Overview 





5. Recommend a preferred Chapter VI Researcher 


solution 


1. Identify Problem 

Within the context of the personal property program, there has been no 
single and certain definition of program characteristics requiring change. While 
senior government officials have repeatedly identified shortcomings in the current 


program, criticism has focused on negative outcomes, not on specific outputs of 


2 





the propram.? In GAO/NSIAD reports 90-50 (1990) and 94-26 (1993), GAO 
recommended that MTMC replace or modify the domestic household goods 
bidding system and that DoD adopt commercial practices, such as using a smaller 
—e of carriers. These problems, a high rate of claims, and general customer 
dissatisfaction with the move process have formed the basis of DoD efforts to 
reengineer the personal property program. This study will not attempt to identify 
specific issues or problems inherent in the personal property program. 

2: Identify Solutions 

As previously noted, there are three separate alternative solutions to the 
problems inherent to the personal property program. These polices do not 
necessarily represent the best set of alternatives, but _—— analysts have found 
them to be plausible responses to the policy problems. Furthermore, the 
alternatives are mutually exclusive and have a sufficiently different focus. In this 
study, research and subsequent analysis will focus on these existing policy 


proposals. 
3: Develop Criteria 
Establishing criteria requires either assumptions or knowledge about the 


situation: policy goals and stakeholder values. Numerous government and 
industry documents provide an initial list of the goals of the various alternatives. 
> Note the previously cited comments of the Secretary of Defense and the congressional testimonies of Mr. 


Roger Kallock, Deputy Under Secretary of Defense (Logistics), Lieutenant Generals Thompson and 
Coburn, and Mr. David Warren, Director of Defense Management Issues, GAO. 
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However, it is necessary to determine how each alternative attempts to solve the 
stated problem. In other words, it is necessary to link the goals of the personal 
property reengineering program to the sub-goals and criteria of each alternative. 
This schematic or map allows the evaluator to develop a set of criteria that can be 
used to evaluate the extent to which each policy alternative meets a specific 


criterion. Chapter III, Methodology, details the criteria development process. 





4. Evaluate Policy Alternatives 

Once the evaluation criteria are established, the analyst.must determine the 
value to be placed on each criterion when evaluating and comparing policy 
alternatives. Value is based on perspective. In this study, the perspective of each 
dominant stakeholder group is assessed. Chapter III, Methodology, describes the 


steps in this assessment. Chapter [V, Data, summarizes the values stakeholders 





place on the goals, sub-goals and evaluation criteria. Chapter V, Data Analysis, 
uses the evaluation criteria and interview results to evaluate each policy alternative 
from the stakeholders’ perspectives. 

5. Recommend Preferred Alternative 

The policy analysis process deals with complex problems and, most 

_ importantly, multiple goals. It is unlikely that any policy is going to be ideal in 

terms of all goals. However, by establishing a single set of evaluation criteria and 
determining the value that dominant stakeholders place on those criteria, it is 


possible to recommend a preferred alternative from a particular perspective. More 


Os 


importantly, it is possible to establish a set of evaluation criteria based on the 
aggregate values of stakeholders. Chapter VI, Conclusions and 
Recommendations, identifies the policy that is optimal for each stakeholder group 


as well an alternative that is in the best interests of all. 


E. CHAPTER SUMMARY 

Policy analysis “is not a specific methodology, but it makes use of a variety 
of methodologies in the context of a generic framework (Walker and Fisher, p. 
1).” It often proves difficult to establish the boundaries and context of the 
problems when evaluating polices in the public sector. Multiple competing 
stakeholders often have varying and conflicting interests and perspectives on the 
goals and measurement criteria of public programs. A spectrum of approaches to 
discerning evaluation criteria exist. A framework for policy analysis is established 
based on the basic components of several models and approaches. This 
framework provides a method for choosing a preferred course of action amid 


multiple complex alternatives 
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Til. METHODOLOGY 





A. INTRODUCTION 

This chapter presents the methodology used to conduct this study. The 
highlighted portions of Figure 6 show the steps and supporting task that are 
covered in this chapter. It describes the selected research focus, the primary 
research methods, and the steps in the evaluation criteria development process 
outlined in apie II. Furthermore, this chapter furnishes a detailed description 
of how the goal and evaluation criteria mapping process is used to develop a single 


set of criteria. An assessment of the extent to which the policy alternatives meet 





the criteria is provided. The chapter concludes with an outline of the stakeholder 
analysis process and how the evaluation criteria development process is linked to 
the data collection and analysis stages of the study. 
B. RESEARCH FOCUS 

The objective of this thesis is to determine which policy alternative for 
reengineering the DoD’s personal property program best satisfies stakeholder 
criteria. The researcher chose to focus on efforts to reengineer the personal 
property program due to the subjective nature of this effort’s goals, the existence 
of multiple competing stakeholders, and the potential impact of an implemented 
policy on viniiueniiibiie: DoD efforts to achieve significant change in the 


personal property program have been hampered by a lack of consensus among 
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stakeholders as to the problems with the program and possible solutions to those 


problems. 
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Figure 6. Overview of Chapter IIT 





Each stakeholder has identified their own set of priorities and goals for any 


alternative to the current personal property program. DoD has attempted to 


represent their own interests as well as the interests of the servicemember in | 
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crafting policy alternatives. Industry has exercised its political and market 
influence in a bid to ensure that their goals and criteria are incorporated into the 
policy that is finally implemented. This trichotomy of stakes has created a need to 
evaluate the various policy alternatives from the perspectives of each dominant 
stakeholder group as well as in ageregate. 


Relevant research and analysis provides a framework to define a single set 





of criteria that can be used to evaluate the current personal property program 
policy alternatives from the various stakeholders’ perspectives. This effort 
employs the basic tools of evaluation research. The design involves collecting 


individual opinions and archival data using primary and secondary sources on 





three alternate proposals to reengineer the personal property program. The process 
of developing evaluation criteria used in this thesis is broken down into four 
iterative steps: 


Identify goals of the personal property program reengineering effort 
2. Establish evaluation criteria 


Assess three policy alternatives developed by DoD for reengineering 
DoD’s Personal Property Shipment and Storage Program 


4. Conduct stakeholder analysis and calculate the values each 
stakeholder group places on evaluation critera. 


C. IDENTIFY GOALS OF PERSONAL PROPERTY PROGRAM 
REENGINEERING EFFORT | 


As aresult of the initial joint DoD/industry working group session, DoD 
and industry agreed to ten goals for the reengineered personal property program 


(GAO/NSIAD-97-49, p. 3). In addition to these goals, government, industry and 
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government contractors have also stated or published goals for reengineering the 
program." Figure 7 summarizes these goals. Although these goals are easily 
understood, they are too vague to lead to agreement that they either have or have 
not been met. Any assessment of the extent to which each proposal meets the 
goals is necessarily limited by the lack of precise definitions of each goal and the 
way to achieve it (GAO/NSIAD-97-49, p. 3). “Moreover, the [government and 
industry program] proposals were written in such a way that did not specifically 
address how each would achieve the stated goals (GAO/NSIAD-97-49, p. 3).” 

In order to discriminate the goals of the three pilots or proposed policy 
alternatives, it is necessary to determine which organization or group has the 
ability to ensure its goals were sustained by the alternate polices. Government 
policy development is not only analytically complex but also intensely political, 
involving many powerful interests and convictions that will attempt to influence 
the goals of any effort to change policy (Bonser, McGregor, and Oster, p. 18). 
DoD initially developed three pilot projects that emphasized their goals for a 
future personal property program. After numerous debates, industry protests and 


GAO reports, DoD received Congressional approval to conduct its pilots pro} ects. 


* Primary sources include speeches by public officials, government and industry congressional testimony, 
GAO reports, and interna! documents and memoranda of the DoD. 
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Figure 7. Goals for Reengineering Personal Property Program 


However, even with congressional approval, DoD still found it necessary to 
recognize the needs of industry. As a result, the various DoD and industry goals 
must be synthesized into concepts or general goals that recognize industry’s 


concerns as well as fulfill DoD’s primary goals to improve quality of life. Three 
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general goals were identified: 1) improve servicemember’s quality of life, 2) 
reduce the total cost of the personal property program, and 3) maintain a 
competitive market and limit the impact of a policy change on small businesses. — 


These goals formed the basis of the forward mapping process to be detailed later. 


D. ESTABLISH EVALUATION CRITERIA 

Evaluation criteria must be able to measure the extent to which the goals or | 
concepts of a policy are achieved. “A good criterion provides a basis for 
measuring progress toward achieving a goal (Weimer and Vining, p. 223).” 
However, not every goal can be reasonably quantified by a single objective or 
criterion. “Rather than emphasize a single objective that only measures one 
dimension of a goal, you should try to specify criteria that cover all the important 
dimensions (Weimer and Vining, p. 223).” As Weimer and Vining point out, 
multiple criteria may be used to measure the extent to which a policy’s goals are 
achieved. “The set of criteria should capture all the important dimensions of the 
relevant goals (Weimer and Vining, p. 225).” 

The evaluator must be able to systematically link the goals of the policy 
with the supporting sub-goals and criteria. There are a number of methods that 
can be used to establish the relationship between goals and criteria. One such 
method is labeled, “forward mapping (Bryson, 1995, p. 287).” Through the use 
of forward mapping and similar techniques the evaluator is able to develop a map 


linking the program’s goals with its supporting sub-goals and criteria. The 
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forward mapping process begins with a program’s goals, with as clear a statement 
as possible of the policy maker’s intent, and proceeds through a sequence of 
increasingly more specific steps to define performance measures (Bryson, p.287). 
At the bottom of the process, one states, again with as much precision as possible, 
the criteria to be used, measured in terms of the ordinal statement of intent 
(Bryson, p. 287). The end result of this effort is the development of a map that 
defines the policy’s goals and the characteristics that support achieving those 
goals. 

Three general goals are defined that embody the intent of the reengineering 


effort. These goals are improve quality of life, reduce total cost, and reduce the 





impact on small business. Sub-goals are identified that support the attainment of 
these general goals. Criteria are established based on these sub-goals. The 
evaluation criteria must be able to measure the extent to which each pilot project 
manifests the goals and corresponding sub-goals. Therefore, each pilot must be 
assessed in order to determine if the criteria are adequate and are representative of 
the scope of the pilot projects. There is no attempt to identify criteria beyond the 
scope of these three alternative policies. The mapping process is useful in 
reducing the complexity of the goals and sub-goals to the point that the 
performance measures or criteria are evident. Evaluation criteria are established at 
the lowest level of indenture. The results of the “forward mapping” process are 


summarized in Figure 6. 
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The adequacy of the criteria is verified using two common research 
diate The first is the collection of data through unobtrusive measures, 
specifically archival research, and the second is the collection and verification of 
- data through electronic correspondence. 

Primary sources are used in this study to identify the goals and sub-goals of 
each policy alternative. Primary sources include speeches, congressional 
testimony, reports, and documents issued by various U.S. government agencies. 
Examples include reports by the General Accounting Office, inter-agency 
information papers, briefs and memoranda, and contractual statements of work. 
Secondary sources are used to clarify specific aspects and characteristics of the 
bevern For example, the terms “best value” and “commercial business 
practices” were used indiscriminately. As a result, it was necessary to define the 
limits of each as it related to the pilot projects. 

Data collected from archival sources served as a basis for the questions 
asked through electronic correspondence. Data were requested via electronic mail 
to verify the accuracy of the information gained from primary or secondary 
sources. Data were collected from a limited group of individuals having expert 
knowledge of the subject area due to their position or experience. Correspondence 
also provided additional information that was not gathered from archival 
resources. This information provided insight into known goals and sub-goals and 


highlighted unknown features of each pilot. 
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Improve Quality of Life. This goal is based on achieving customer satisfaction and 
improving or expanding services to the customer. 


A. Use of Federal Acquisition Regulation (FAR)-based contracting method 


B. Loss and Damage Claim 
1. Full Replacement 


2. Direct Settlement with: 
a. 3" Party 
b. Transportation provider 
c. Government 
3. Reduced claim preparation requirements 
4. Reduced claim settlement period 
C. a Qualification - Standards for contract award 
Financial stability - providers have strong financial credentials, low risk of 
contractor failure 
2. Ensured capacity - proven track records and capability proportionate to the 
shipper’s requirements. 
3. Past Performance - imposition of stringent performance standards 
e On-time pick-up and delivery rates 
e Loss and damage rates comparable to or better than the lowest commercial 
account’s loss and damage rates 
e Customer feedback (intangibles) - 1-e., professional crews, comfort with 
provider 
4. Inclusion of Service Contract Act - this Act requires the Providers to pay 
employees the prevailing minimum wage for the area in which they are working 
5. Inconvenience payments - provider pays servicemember in the event that the 
provider is unable to deliver personal property when scheduled 
D. Provider Performance Measurement - Relationship with providers contingent on 
them meeting and maintaining performance standards 
Use of customer surveys 
Random inspections 
Periodic performance assessment 
E. Move Management 
Pre-move counseling 
Single point of contact move management 
Binding estimates for excess costs 
Destination Information 
Shipment tracking 
Delivery coordination 
Real Estate assistance 


SO 


Figure 8. Goals and Evaluation Criteria 


a 


I. Reduce Total Cost. This concept includes direct (transportation, storage, accessorials) 
and indirect cost. The indirect baseline cost includes management of the personal property 
program, personal property services provided, quality assurance, financial transaction 
processing, and additional systems operations and maintenance cost. This concept is based on 
reducing the government’s administrative burden through eliminating or improving current 
processes or implementing better business practices. 

A. Pricing Strategy -Improve rate solicitation process and adopt commercial or standard 
rate structure 
1. Continued use of DoD rate solicitations, domestic and international 
2. Adoption of commercial tariff, domestic 
3. GSA Tender 1-W, domestic and international 
4, Single-Factor-Rate (SFR), domestic and international 
5. Commissions - paid to move managers by transportation providers 
B. Traffic Allocation - Allocation of traffic to and amongst providers 
1. Contract consolidation - consolidation of 2 or more procurement auenients 
for goods or services previously provided or performed under separate small 
contracts into a solicitation of offers for a single contract 
2. Traffic distribution based on quality 
3. Traffic distribution performed by: 
a. 3" Party 
b. Servicemember 
c. Government 
4. Reduction or elimination of current use of the Traffic Distribution Record 
(TDR) 
5. Fewer contracts awarded 
6. Longer contract periods 
C. EDI Strategy: EDI capability or a willingness to do business in an EDI environment 


D. Reduction in the complexity of the Movement Category and Service Structure 


Il. Reduce Impact on Small Business. This concept will be measured against the Small 
Business Act, Public Law 85-536, as amended, which states that the government-wide goal 
for participation by small business concerns shall be established at not less than 23 percent 
revenue. Small business is defined for purposes of this evaluation plan as those businesses 
with annual average gross revenue of $18.5 million or less (prime contractor or Provider). 
This concept strives to develop and maintain a competitive market. 

A. Traffic Structure: combines traffic flow and traffic scope-focus on streamlining 
existing network based on outbound traffic lanes from an area of responsibility 
(AOR) toarate area (RA) 

B. Payment | 
1. Defense Finance & Accounting Service (DFAS) 

2. Bank Service Contract 
3. Government Credit Card: IMPAC 

C. SBA Compliance Means 
1. Application of Small Business Act 
2. Small Business Set Asides as Primes 


Figure 8. Goals and Evaluation Criteria (cont.) 
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The use of archival sources and electronic correspondence has certain 
limitations. First, archives may contain errors, be incomplete or be no longer 
relevant. Secondly, the source may present only one perspective. Finally, 
different meanings may be assigned to similar terms or concepts based on the 
archival source or respondent. These problems were mitigated by using up to date 
multiple sources. Multiple sources from different levels and components of DoD 
and government were reviewed. Furthermore, archival sources and electronic 
responses were cross-referenced. For example, congressional testimonies were 
checked for consistency with GAO reports. Also, electronic responses were used 
to verify accuracy and relevancy of internal memoranda, statements of work, and 


performance work statements. 


E. ASSESS POLICY ALTERNATIVES 

This step determines the extent to which each policy alternative fulfills the 
established criteria. The results of the forward mapping process summarized in 
Figure 8 were used to develop criteria by which to assess the policy alternatives. 
There are two steps in this process. The first step is the identification of general 
goals and the specification of sub-goals. In the second step, criteria were 
developed to measure the extent to which the goals and sub-goals are achieved. 
Additionally, the policy alternatives were compared and appraised at the criteria 


and sub-criteria level where data were available. 
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Thus, the forward mapping process creates a strong relationship between 
the goals, supporting sub-goals and the evaluation criteria. For example, in Figure 
9, there are five ways to improve quality of life for the servicemember: A) 
contracting method, B) loss and damage claims, C) provider qualifications, D) 
provider performance measurement, and E) move management. There are three 
criteria for measuring the extent to which provider performance measures have 
been established: 1) customer surveys, 2) random inspections, and 3) periodic 
performance measurement. 

Once the goals and criteria were identified and their relationship to each 
other was established, I created a scale to esac the extent to which each policy 
alternative embodied the evaluation criteria. The following scale was used to tate 


how important the criteria were to the pilot project: 


J = Not important 

2 = Little importance 

3 = Medium importance 
A = High importance 

5 = Critical 


Using this scale, I independently rated each pilot project with respect to the 
criteria. I made a judgment as to the value each policy alternative places on a | 
criterion. The pilot project characteristics displayed in Figure 2 formed the basis 
of this assessment. Statements of work and performance work statements were 


also used to provide details on specific aspects of each pilot. 
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Goal ——“‘ws=sd| Sub-goal————“‘i‘C*:«CCrrritterriaa 
A. Contracting Method FAR -based 


1. Full Replacement 





Sub-criteria 7 






a. Move Manager 
2. Direct Claim Settlement: b. Transportation 
Provider 
c. Government 






B. Loss and Damage Claim 


3. Reduced claim preparation 
requirements 

4. Reduced claim settlement period 
1. Financial stabili 

2. Ensured capaci 

3. Past performance 

4. Inclusion of Service Contract Act 
5. Inconvenience payments 


1. Customer Surveys 
D. Provider Performance 2. Random Inspections 
Measurement 3. Periodic Performance Assessment 


1. Pre-move counseling 

2. Single point of contact move 
management 

3. Binding estimates for excess costs 
4. Destination information 

5. Shipment tracking 

6. Delivery coordination 

7. Real Estate assistance 

1. Continued use of DoD solicitations 


C. Provider Qualification 









I. Improve 
Quality of Life 














E. Move Management 


Domestic 
International 
Domestic 
Domestic 
International 
Domestic 
International 


2. Commercial Tariff 
3. GSA Tender 1-W 





A. Pricing Strategy 







4. Single-Factor-Rate 


5. Commissions for move manager 
1. Contract consolidation 
2. Traffic distribution based on quali 





IJ. Reduce 
Total Cost 







a. Move Manager 
b. Servicemember 
c. Government 


3. Traffic distribution performed by: 
B. Traffic Allocation 

4. Reduction or Elimination of Traffic 

Distribution Record 

5. Reduction in Contracts awarded - 

6. Increase length of Contract award 


C. EDI capability required 
D. Service Structure Reduction in movement and storage 
categories 

WI. Limit A. Traffic Structure Change current structure of AOR to 
Impact on 
Smal] 
Business; 


1. DFAS 

2. Bank Service Contract 

3. Government Credit Card 

1. Application of Small Business Act 
2. Small Business prime contractor set 
asides 


B. Payment 
Maintain a 
Competitive 


Market C. SBA Compliance Means 





- Figure 9. Goals and Evaluation Criteria 


Figure 10 lists the values assigned for each criterion relative to the pilot 
projects. These values represent how important each criterion is to a particular 
policy alternative. For example, the SAM pilot places medium importance on the 
use of FAR-based contracts. On the other hand, FAR-based contracts are highly 
important in the FSMP and critical to the MT'MC pilot. 

Interaction through electronic correspondence with the project officers for 
each pilot project biased this assessment somewhat. Certain project officers were 
better able to articulate the characteristics and benefits of their pilot. In an attempt 
to reduce this bias, I limited my interaction with the project officers. This step is 
performed prior to determining stakeholder values in order to prevent stakeholder — 
comments from further biasing this study. 

The lack of inter-rater reliability also creates a bias in this assessment. 

Only one rater was available to conduct the assessment of the policy alternatives. 
This limitation is aresult of the complex nature of the personal property pro gram 
and the reengineering effort. In order to rate the pilots, a level of knowledge and 
familiarity with the program is required. The use of multiple raters in follow-on 
research will provide a greater degree of accuracy and reliability to this 


assessment. 
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Sub-goals Evaluation Criteria Pilots 


[SAM_[FSMP_| MIMC_ 
Contracting Method |FARbaed SOSC=~“~*~*~*—“—s—*S*S*S*~*—“—‘—‘—‘—‘Ss~‘“‘~‘<‘dSSZSCdY 
Loss and Damage | FullReplacement —=S~CS~—“—SSC—SCSCS ST 
Claim Direct Claim Settlement | Move Manager——=—SsdT SST SS 
Transportation Provider [| 5_| 1 


Reduced claim preparation requirements 
Reduced claim settlement period 
Financial stability 

Ensured capacity 

Past performance 

Inclusion of Service Contract Act 
Inconvenience payments 

Customer Surveys 

Random Inspections 

Periodic Performance Assessment 
Pre-move counseling 

Single point of contact move management 
Binding estimates for excess costs 
Destination information 

Shipment tracking 

Delivery coordination 

Real Estate assistance 

Pricing Strategy Continued use of DoD | Domestic 
solicitations International 
Commercial Tariff Domestic 
GSA Tender 1-W Domestic 
International 
Domestic 
International 
Commissions for move manager 
Contract consolidation 


Traffic distribution based on quality | 4 | 4 | 3. | 
2 |Move Manager 


Traffic distribution Move Manager 


performed by: [Servicemember.—~+(|~4——~d| Sr 

2 

[Reduction or Elimination of Traffic Distibution Record | 3 | 5 [| 3 

[Reduction in Contracts awarded ——SSSSC=~—S 

increase length of Contract award —=—S~=~—~—“‘—~—*é‘iSCSCS 

EDIeapabiltyrequred SSOSC—~—“—SCSC“sSC“—“‘sSCSs‘“‘“‘iTSSC‘dYS OS 

[Service Structure | Reduction in movement and storage categories _—+4|~—=3~~=«|~—S«dT Sid] Sd 

Traffic Structure | Change current structure ofAORtoRA—~—=S—=~sSC“‘iSC‘T'S} 
Payment 5 

"Bank Service Convact ——SSCSC~—~—“‘“—S*“‘“‘~‘~*S*~*~—‘“iCS*iaSC(C‘“RSSCS 

[Government Credit Card ——SOS—~—~—SCS—CSsCSCSSCSSS 

SBA Compliance | Application of Small Business ——=~S=~—~—*S*—*—~—sSC‘SSC*dTCSCi TC 

Means [Small Business prime contractor setasides——=—=S«dESiSP 


Figure 10. Value of Criteria for Each Pilot Project 
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F. STAKEHOLDER ANALYSIS PROCESS 

Once the evaluation criteria are determined, the evaluator must establish the 
value to be placed on each criterion. Attempts to introduce policy analysis into 
most arenas of public policy inevitably founder on evident disagreements over 
both the measurement techniques and value criteria (Fischer and Forester, p.232). 
In undertaking their work, evaluators usually find themselves confronted with 
individuals and groups who hold competing and sometimes combative views on 
the policy alternatives and whose interests will be affected by the outcome (Rossi 
and Freeman, p.422). These individuals and groups often place vastly different 
values on evaluation criteria. Smith points out that “it is rare to find consensus 
about what constitutes the objectives of a public sector program (Smith, 1996, 
—/p.9).” 

In order to establish whether consensus exists or, when no consensus exists, 
the relative weight or value to assign each criterion, the evaluator must perform a 
- stakeholder analysis. The identification of key stakeholders and the determination 
of a set of stakeholder criteria are critical to choosing a preferred alternative from 
amongst complex policy options. “If an organization does not know who its 
stakeholders are, what criteria they use to judge the organization, and how the 
organization is performing against those criteria, there is little likelihood that the 
organization will know what is should do to satisfy its key stakeholders (Bryson, 


p. 70).” The stakeholder analysis process is broken down into two stages for the 
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purposes of this study: 1) identifying the stakeholders and 2) determining the value 





they place on each objective. 

1. Identification of Stakeholders 

The first step in a stakeholder analysis is to identify an organization’s 
stakeholders. This step usually begins with the construction of a stakeholder map 
(Roberts and King, 1989, p. 65). Mitroff defines stakeholders as “all those parties 
who either affect or are affected by an organization’s actions, behavior, and 
policies (Mitroff, 1983, p. 4).” This definition is very similar to the definition 
used by Freeman. Freeman defines stakeholders as “any group or individual who 


can affect or is affected by the achievement of an organization’s objectives 





(Freeman, 1984, p. 39).” Both definitions convey essentially the same meaning 
and are used interchangeably in this study. 

The purpose of the stakeholder map is to identify the stakeholders in the 
reengineering of the personal property program. An initial stakeholder map must 
not leave out any group or individual that can affect or 1s affected by an 
organization’s policies (Freeman, p. 52). Roberts and King recommend reducing 
the number of stakeholders by categorizing them into larger, more general groups 
(Roberts and King, p. 69). Once identified, individual stakeholders are grouped 
according to similar stakes and organized into an overall “stakeholder map” 


(Roberts and King, p.69). 
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Three stakeholder groups relevant to reengineering the personal property 
program were identified. Figure 11 contains a list of the dominant stakeholder 
groups. These groups are 1) government, 2) industry, and 3) servicemembers. 
Industry is broken down further — two sub-groups: relocation management and 


transportation providers. 





























Government Industry 






e OSD 1. Transportation 
e USTRANSCOM Providers 

e MTMC 2. Relocation 

: sn Personal Property Managers 


Program 
Reengineering Effort 


Servicemember 


Figure 11. Stakeholder Map 


a. Government Stakeholders 

The United States Government is responsible for initiating and 
funding the program as well as for deciding which policy alternative is instituted. 
There are numerous stakeholders within the government. The primary stakeholder 
is the Department of Defense. Agencies and components of DoD are also 


classified as stakeholders to the reengineering effort. 
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b. Industry Stakeholder 





oe stakeholders are commercial organizations and groups who 
compete for business under the personal property program. There are two major 
sub-groups within the industry stakeholder group. These are 1) relocation 
management ee and 2) transportation providers. 


(1) Relocation Managers. Relocation managers provide a 





wide range of relocation services through an integrated network of transportation 
providers, real estate agents, and move coordinators. The Military Mobility 
Coalition is an ad hoc organization formed in the spring of 1996 to represent the 


relocation management industry’s interests in working with the DoD’s household 





goods reengineering initiatives (Groover, Jr., 1999). The majority of membership 
consists of relocation nv move management companies. The coalition currently 
includes twenty-seven military organizations, Business Executives for National 
Security, and most major real estate networks (Groover, Jr., 1999). - 

(2) Transportation Providers. Two organizations represent 
and promote the interests of the transportation and storage portions of the moving 
industry before federal and state legislative and regulatory bodies. These are the 
Household Goods Forwarders Association (HGFA), and the American Moving 
and Storage Association (AMSA).° The members of the HGFA comprise all 


facets of the moving industry, including moving and storage firms, national van 


> The American Moving and Storage Association was formed when the American Movers Conference and 
the National Moving and Storage Association combined. 
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lines, line-haul firms, international and domestic forwarders, port agents, claims 
adjusters, material providers, insurance providers, technology firms, and others 
(Head, 1999). Members include some 1,500 international and domestic 
companies, the majority of which are small businesses (Head, 1999). The AMSA 
is a national trade association of the moving and storage industry. It has 
approximately 3,400 members nationwide representing the entire spectrum of the 
industry and including national van lines, their affiliated agents (carriers and non- 
carriers) and independent regional and national carriers (Harrison, 1999). 

C. Servicemember 

Servicemembers are the customers of the personal property program. — 
There are over 1.4 million servicemembers in the United States Armed Forces 
serving in 140 countries worldwide. There are an additional 1.9 million family 
members. The majority of servicemembers are under the age of twenty-five years 


old and have served less than 10 years. 


2. Determining Values Stakeholders Place on Evaluation Criteria 

The second step of the stakeholder analysis is to determine how important 
an objective is to each stakeholder and respective stakeholder group. “Different 
stakeholders do not generally share the same definitions of an organization’s 
‘problems,’ and hence, they do not in general share the same ‘solutions (Mitroff, 


p. 5).’* Therefore, an evaluator must strive to determine the value stakeholders 
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ince on the objectives, and the associated measurement criteria, that formulate a 
given “solution” or policy. 

Conflict over values arises because the maximization of one value 
inevitably is at the expense of another (Fischer and Forester, p. 232). Evaluators 
are often unsure whose perspective they should take in designing an evaluation 
(Rossi and Freeman, p. 425). As noted in Chapter II, there is no proper 
perspective, but rather different perspectives may be equally legitimate. The 
responsibility of the evaluator is not to take one of the many perspectives as the 
legitimate one, but rather to be clear from which perspectives es 
evaluation is being undertaken (Rossi and Freeman, p. 425). 

In many cases the information required to complete this step is readily 
available from archival records. “In some cases, however, the information must 
be syseniatioally gathered by interviewing individual stakeholders (Freeman, p. 
114).” Archival sources provided background on the perspectives of the various 
stakeholder groups. However, interviews were necessary to determine the value 
stakeholders place on the evaluation criteria. 

a. Developing a Questionnaire 

Once the initial stakeholder analysis was completed, I developed a 
questionnaire to ascertain the value that stakeholders place on the evaluation 
criteria. There is no attempt to identify stakeholder objectives that fall outside the 


scope of the three alternative policies. A copy of the questionnaire is provided in 
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Appendix B. Questions were phrased in such a way as to necessitate a structured 
response from each interviewee. The questionnaire provides a single format by 
which to evaluate responses from the sample. 

b. Choosing a Sample 

In order to conserve time and money, a convenience sample was 
used. Interviewees were chosen from each stakeholder group. A total of 
seventeen interviews were conducted with representatives from the three 
stakeholder groups. Seven interviews were conducted with representatives from 
the government, four interviews were conducted with members of industry, and 
six interviews were conducted with servicemembers or servicemember 


representatives. 


Interviews were conducted with government officials at every level 
of the reengineering effort beginning with USTRANSCOM and ending at the 
project management level. Interviews with members of industry were divided 
equally between transportation providers and relocation managers. All 
servicemembers interviewed were officers in the United States Marine Corps with 
six to fifteen years of active service and having moved an average of seven times. 


A representative of a servicemember organization was also interviewed. 
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Cc. Conducting the Interviews 


Interviews were conducted telephonically. Each interviewee was 
told the goal of the interview. The interviewee was then asked how important they 
felt each criterion was to any effort to change the current personal property 
program. The respondents answered the questions using the following scale: 

0 = Not applicable 

1 = Not important 

2 = Low importance 

3 = Medium importance 
4 = High importance 

5 = Critical 

Definitions for the evaluation criteria were based on the goals and 
criteria described in Figure 8. During the course of the interviews, respondents 
often provided additional comments and information beyond the scope of the 
question. This information was useful in developing a more thorough 
understanding of the issues involved in the reengineering effort and provided 
insight into the political dynamics of public policy evaluation and implementation. 
These comments are discussed in Chapter VI, Conclusions and Recommendations. 

d. Classifying Results 

Collected data were coded into a spreadsheet and measures of 


central tendency were calculated for each evaluation criterion, including arithmetic 
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means and standard deviations. Using the measures generated from the data, 
additional measures were compiled at the sub-goal and goal level for review and 
examination. This data is presented in Chapter IV. An analysis of the results of 
the statistical results is provided in Chapter V. 

é. Validation 

If the evaluation process is to be enhanced, the data gathered must be 
valid; that is, the “right” responses must be assessed, and in a manner that will 
elicit meaningful measurements (Levine, Berenson, and Stephan, 1998, p. 9). 
Interview data must be validated as far as possible to ensure the accuracy of the 
results. Misunderstandings may be introduced because of difficulties in 
establishing proper modes for communicating with different stakeholders (Rossi 
and Freeman, p. 425). Stakeholders define terms in different fashions and their 
responses are based on that definition. Questions were to provide each 
interviewee with same definition of each criterion. However, interviewees had 
different degrees of involvement and familiarity with the alternative policies and 
characteristics of the personal property program. In summary, although the 
methodology was to apply standard definitions to all the criterion, interviewees 
often required additional clarification beyond the standard definitions. 

Another potential problem is the reliability of the results. Bryson 
points out that “stakeholders may not be completely honest (Bryson, p. 73) 


Responses are the interviewees’ professed or espoused values, not necessarily the 
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true value they place on the criteria. Value is also relative amongst respondents. 





Although the scale was defined, my speculation is that some respondents may 
have inflated their values while others deflated theirs. This conclusion is based on 
the general comments some interviewees made with respect to the criteria. In 
these cases, I reminded the interviewee of the scale being used. Sometimes 


interviewees attempted to answer according to the program they supported and 





sometimes they espoused their personal opinions. 
An attempt was made to mitigate potential sources of error by 


verifying data with archival sources. Responses were double-checked to ensure 





consistency with congressional testimony, published stakeholder documents and 
GAO reports. [ also reviewed responses to identify any obvious disparate trends 


amongst stakeholders within a stakeholder group. 


G. CHAPTER SUMMARY 

The four-step method to develop evaluation criteria and establish the value 
stakeholders place on each criterion is an iterative process. Evaluation criteria are 
developed using data gathered through archival and electronic correspondence 
research. While many steps in the process are largely subjective, a concerted 
effort was made to crosscheck and verify each assumption. The result is a list of 
evaluation criteria and an assessment of the extent to which each pilot project 


meets those criteria. The value stakeholders place on each criterion is established 
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by conducting interviews with a convenience sample of stakeholders. This data is 


presented in Chapter IV. 
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IV. DATA 


A. INTRODUCTION 

This chapter presents and discusses the data collected on the values 
stakeholders place on the evaluation criteria for reengineering DoD’s personal 
property program. The data presented were gathered through telephonic 
sateaviens using a standard questionnaire. The respondents were from a 
convenience sample of stakeholders. A total of seventeen interviews were 


conducted. Every stakeholder contacted agreed to be interviewed. 


B. STATISTICAL MEASURES 

Interviewee responses were recorded for each evaluation criteria. The data 
were then entered into a spreadsheet. The arithmetic mean and the standard 
deviation of individual stakeholder responses were computed for each stakeholder 
group as well as an aggregate for all stakeholders. Summarized data reveal 
differences in values between stakeholders, as well as expose shared values 
amongst stakeholders. Interview data and statistical analysis is presented in 
Appendix C. 

Data are presented and discussed at the goal and sub-goal level as shown in 
Figure 9. I integrated evaluation criteria data at the sub-goal level by calculating 
the mean of the group averages for the criteria that support each sub- goal. For 
example, the value assigned to “D) performance measurement” by each 


stakeholder group represents the average of the values calculated for: 1) customer | 
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surveys, 2) random inspections, and 3) periodic performance assessment. I 
integrated sub-goal values at the general goal level by computing the average for 

| ‘the sub- goal values that support each goal. For instance, I derived the value for “I) 
improve quality of life” by determining the average for the following sub-goals: 
A) contractor method, B) loss and damage claim, C) provider qualification, D) 
performance measurement and E) move management. 

Statistical methods help reduce the “raw” data generated from the nieeion 
process by generating summary measures. The measures help to focus the data 
and reduce the variation between individual stakeholders within a stakeholder 
group. The use of common statistical measures simplifies the evaluation process . 
when numerous stakeholders value multiple complex evaluation criteria 
differently. However, an arithmetic mean may be skewed when extremely high or 


low responses known as outliers are present in the data. 


1. Improve the Servicemember’s Quality of Life 


Improve Quality of Life 





Of the three goals, stakeholders place the highest value on improving 


servicemembers quality of life. Clearly, this goal is seen as the key component in 
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the effort to reengineer the personal property program. Relocation providers place 
the highest value on this goal. Transportation providers place the lowest value on 

this goal. Government and servicemember stakeholders are the most consistent in 
placing a high value on this goal. 


a. Contracting Method 


‘Stakeholder = =. '|- Average. 
Providers | ~1.00 | 0.00 























Transportation providers are diametrically opposed to the use of 
FAR-based contracts in the personal property program. Government and 
relocation management stakeholders consistently support the use of FAR based 
contracts with the exception of one individual stakeholder. Servicemembers, for 
the most part, are not concerned with the type of contracting method. 


b. Loss and Damage Claim 


| Stakeholder | Average" | Std. Dev. _ 
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There is generally strong support amongst stakeholder groups for 
improving the loss and damage claim process. Providing full on eo 
coverage for the servicemember during the move received the highest aggregate 
value and can be considered critical to any effort to reengineer the personal 
property program. The standard deviation indicates a large variation in responses. 
This variation is due to the choices respondents had when considering direct claim 
settlement. Interviewees were asked to choose between three mutually exclusive 
options when settling a loss or damage claim. The majority of stakeholders prefer 
claims settlement with the transportation provider. A reduced claim preparation 
requirement and claim settlement period are both highly valued by all 
stakeholders. 


C. Provider Qualification 


Provider Qualifications 








st : a a ise Soe eae 2 Swe Tate 
aa Paras Pera ee mae aecar oP ise, faa POOR ee ea 







Creating or increasing standards for awarding contracts is valued as 
having medium to high importance to the stakeholders. Stakeholders also perceive 
ensuring industry has the capacity to meet contractual obligations and qualifying 


providers based on past performance as important features to the reengineering 
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effort. The inclusion of the Service Contract Act is seen a having very little value 
by industry. 


d. Performance Measures 


‘ho i a —— | Stdz Dev. : 




























Stakeholders are fairly uniform in their feeling that there should be 





some form of performance measurement within the personal property program. 
Stakeholders value the use of customer surveys the most. However, periodic 
performance assessments are almost as highly valued. 


é. Move Management 


Stakeholder = Paverge [Si 1. Dev. 


















This sub- goal includes the provision of expanded services to the 
servicemember by a move manager or transportation provider. Government and 
the relocation management stakeholders place the highest value on expanding 
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move management services. Servicemembers and transportation providers feel 
that as a whole these services are moderately important. A single point of contact 
during the move process, shipment tracking, and delivery coordination are 
perceived as providing the most value an aggregate of stakeholders. While real 
estate assistance is seen as critical to improving customer satisfaction by the 
relocation industry, the majority of other stakeholders place very little value on 
this criterion. 


2. Reduce Total Cost of the Personal Property Program 


Reduce Total Cost 


Aggregate Average _| Std. Dev. 
7 





The goal of reducing the total cost of the personal property program is 
moderately valued by the stakeholders. Stakeholders place the greatest value on 
changing the traffic allocation process and encouraging the use of electronic data 
interchange within the personal property program. Stakeholders have strong 
feelings on the type of pricing strategy used in the policy alternatives. However, 
‘most stakeholders were ambivalent about changes to the category and service 
structure. Servicemembers demonstrated a discernible lack of interest in efforts 


supporting the reduction in total cost. 
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a. Price Strategy 


Stakeholder © ay : 
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Stakeholders are unanimous in their opposition to a continuation of 
the current DoD solicitation process. Moreover, servicemembers do not see 


pricing strategies as their concern. The majority of stakeholders support the use of 





a domestic commercial tariff and an international single factor rate. 
Transportation providers support the use of a commercial tariff, but resist efforts 
to deeply discount the commercial rate. Transportation providers oppose 
commissions for move managers as a violation of the Anti-Kickback Act.® 
Government stakeholders and move managers, on the other hand, believe | 
commissions are representative of common commercial practices and should be 


allowed in the personal property program. 


© The Anti-Kickback Act defines a kickback as anything of value provided to a prime contractor for the 
purpose of improperly obtaining favorable treatment in connection with a government subcontract. 
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b. Traffic Allocation 


Traffic Allocation 





‘Stakeholders place a high value on using quality as a basis for 
assigning traffic to carriers. However, many stakeholders stated that traffic should 
not be allocated on quality alone, but should be based on “best value.”’ Best value 
describes a process for allocating traffic based primary on quality, but also 
considers cost a factor. The transportation industry views contract consolidation 
and a reduction in contract awards as anti-small business. Government 
stakeholders and relocation managers feel these two criteria are Gonifeni ways to 
reduce administrative requirements and reduce costs. The majority of 
stakeholders support a policy in which either a move manager or the 
servicemember performs traffic distribution. Transportation providers are in 
opposition to any policy that allows move managers to choose carriers. There is a 
consensus amongst stakeholders on increasing the length of contract awards to at 


least one year. 


’ “Carriers” include both motor carriers and freight forwarders 
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Cc. Electronic Data Interchange (EDI) Capability 


Aggregate «|= | 
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Government and industry stakeholders place a relatively high value 
on encouraging the use of electronic data interchange within the personal property 
program. Servicemembers view EDI as moderately important to the reengineering 
effort. Government believes EDI is critical to achieving future cost savings. 


Industry stakeholders see EDI is a viable goal, but are concerned that requiring a 





level of EDI capability may place an excessive burden on small businesses. 


d. Reduce Complexity of Movement Category and Service 
Structure 


Category & Service Structure 


Stakeholder 





Servicemembers 


Stakeholders place only a moderate value on reducing the 
complexity of the movement category and service structure. Relatively, most 


stakeholders viewed this criterion as a low priority. In consonance with this 
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viewpoint, the policy alternatives minimize changes to these categories and 
services. 


3. Maintain a Competitive Market 







Maintain a Co 
Stakeholder =~... | Average | 


Yet 


























Government and industry stakeholders value the contributions of 
small businesses to the personal property program highly. However, there are 
differing opinions on the role small business should play in the program. 
Transportation providers place a high value on limiting the impact of policy 
changes on small business. They have lobbied congress strongly on this point. | 
Government stakeholders and move managers assert that current laws protect 
— businesses and the potential impact on small business should be only 


moderately valued. Servicemembers expressed very little interest in this goal. 
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a. Simplified and Reduced Traffic Structure 
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The general agreement amongst stakeholders is that the traffic 
structure should be simplified and reduced. There are currently over 17,000 traffic 
lanes used by the personal property program. While none of the current policy 
alternatives emphasize this criterion, each alternative has attempted to streamline 
the existing network in some manner. 


b. Payment Options 


‘Stakeholder «|. Average. | Std. Dev. 








3.08 








Many stakeholders feel that DFAS should be the best option. 
However, due to DFAS’ poor performance record, most recommend using a 
Government Credit Card or a Bank Service Contract. Government stakeholders 
are wary of the significant administrative burden caused by the use of the 
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Government Credit Card. Many stakeholders indicated that they support the use 
of a bank service contract not because it provided the greatest value, but because it 
is the least objectionable option. 


C. Small Business Incentives 


_ Small eee Incentives 
Aggregate 












There was a large range of values placed on limiting the impact of a 
policy change on small businesses. Government and the relocation industry view 
small business as a critical component of the moving industry’s capacity. 
However, they feel that small businesses can be integrated into a network of | 
carriers and freight forwarders to achieve greater service and less cost. Many 
small businesses have exclusive relationships with national carriers. This has 
caused widespread cynicism among government stakeholders when denoting 
which businesses are “small” and which are not. Government and relocation 
industry stakeholders feel that meeting small business requirements under the 
Small Business Act are sufficient for ensuring small business participation in the 


personal property program. Transportation providers disagree with this view and 
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support the introduction of policy features that limit or dispel the authority of the 


move manager and increase the participation of small businesses in the program. 


C. CHAPTER SUMMARY 

The interview process requires the collection, coding and reduction of the 
data into a meaningful framework. Data reduction refers to the process of 
selecting, focusing, simplifying, abstracting, and transforming the “raw’’ data that 
are generated from the interviews (Roberts and King, 1996. 250). Coding data 
allows for the interpretation and integration of the results. The valuation of the 
evaluation criteria by the stakeholders provides a basis for the evaluation of the 
policy alternatives from the each stakeholder group’s perspective. Chapter V, 
Data Analysis, will evaluate the alternative policies for reengineering DoD’s 
personal property program using a single set of evaluation criteria derived from 


the stakeholder analysis. 
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V. DATA ANALYSIS 


A. INTRODUCTION 

The purpose of this chapter is to evaluate the alternative policies for 
reengineering DoD’s personal property program using a single set of evaluation 
criteria derived from a stakeholder analysis. Figure 12 displays the steps and 
supporting tasks that are used to analyze the data summarized in Chapter IV. 
Statistical standards and methods are employed to compare and contrast the three 
alternatives. The result of this process is the identification of a policy alternative 


or alternatives that best satisfy stakeholder criteria. 


_ Data Analysis Overview 


| Where? | © Figure? "[° °°. Method? ree oes: 


What? 


1. Determine extent to which each policy Chapter V | Appendix D | Absolute value of difference between 
alternative fulfills values stakeholder . stakeholders’ and alternatives’ values 
groups place on criteria | is calculated 


2. Determine extent to which each policy | Chapter V | Appendix D | Sum differences for sub-goals and 
alternative satisfies stakeholder goals 

expectations as a whole 

3. Determine the policy alternative that Chapter V_ | Figures Compare and contrast total 

best satisfies each stakeholder groups’ mat 13-18 differences 


criteria 





Figure 12. Data Analysis Overview 


B. EVALUATION OF POLICY ALTERNATIVES 

The policy alternatives were evaluated from each stakeholder perspective as 
well as an aggregate perspective using a single set of criteria. Interview data were 
used to generate values for the evaluation criteria. The arithmetic mean of 


individual stakeholder responses represents the value a stakeholder group places 


69 


on a criterion. I then made a comparison between the mean for each criterion and 
the value assigned to a policy alternative shown in Figure 9. 

The absolute value of the difference between the stakeholder group’s 
expressed values and the value I assigned to each policy alternative was calculated 
from this comparison. The difference between these two values indicates the 
degree to which a policy alternative fulfills stakeholder expectations for that 
criterion. The lack of a difference indicates that the stakeholder’s expectations 
have been more fully met. Thus, the greater the difference, the greater the degree 
of incompatibility between the stakeholder values and the policy alternative. 

For example, government stakeholders placed an average value of “3.33” 
on the use of FAR-based contracts. I assigned a value of “5” to this criterion for 
the MTMC Pilot. The absolute value of the difference between the stakeholder 
group’s expectations and the value I assigned to the MT'MC pilot 1s “1.67 (5- 
3.33=1.67).” The SAM pilot, on the other hand, has an assigned value of “3” for 
this criterion. Therefore, the difference in this case would be “.33.” Thus, the 
SAM pilot comes closer to meeting government stakeholder expectations for the 
use of FAR-based contracts. 

1 Evaluation of Aggregate 

The final step of the evaluation process is to determine the policy 
alternative that best satisfies the stakeholder’s criteria as a whole. This was 
accomplished by totaling the differences between a stakeholder group’s 


expectations and a policy alternative’s values. The total difference that is closest 
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to zero designates the policy alternative that best satisfies stakeholder 


expectations. Figure 13 presents the results of the evaluation for the aggregate of 
the stakeholder groups. Overall, the FSMP best satisfies stakeholder criteria. This 
is in a large part due to the move management services that the FSMP provides to 
the servicemember. The FSMP consistently fulfills stakeholder values to a greater 
extent than the other alternatives for the majority of sub-goals. Each policy 
alternative meets stakeholder expectations for reducing total cost and maintaining 
a competitive market to a similar extent. The largest disparity amongst 
alternatives in maintaining a competitive market is for payment options. The 
FSMP proposes the use of a bank service contract as a payment option. The SAM | 


uses a government credit card, and the MTMC pilot continues to use DFAS. 
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Figure 13. Aggregate Total Difference 
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Z: Evaluation of Government Stakeholder Group 


Figure 14 presents the results of the evaluation for the government 
stakeholder group. Overall, the FSMP best satisfies government stakeholder 
criteria. The values government stakeholders place on the sub-goals supporting 
improving the quality of life for servicemembers are congruent with the FSMP. 
However, the alternatives are nearly equivalent in meeting government 
stakeholder values for loss and damage claims. The MTMC pilot does as well as 
the FSMP in meeting the government’s criteria for reducing total cost. 
Furthermore, the SAM pilot meets the government’s criteria for maintaining a 
competitive market to a similar degree as the FSMP. However, the FSMP comes 
closer than the other alternatives to meeting government expectations for both 


reducing total cost and maintaining a competitive market. 
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Figure 14. Government Total Difference 
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3. Evaluation of Transportation Provider Stakeholder Sub-Group 





Figure 15 presents the results of the evaluation for the transportation 
provider stakeholder sub-group. Altogether, the SAM alternative best satisfies the 
transportation providers’ criteria. This is due, in part, to the extent to which the 
SAM pilot fulfills transportation providers’ expectations for reducing total cost 
and maintaining a competitive market. In particular, the SAM pilot does better 
than the other two alternatives in meeting the transportation providers’ criteria for 
allocating traffic and supporting small businesses. The MTMC pilot alternative 
best fulfills the values transportation providers place on apnovii customer 
satisfaction. The MTMC pilot does well in fulfilling stakeholder criteria for loss 


and damage claims and provider qualifications. 
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Figure 15. Transportation Providers Total Difference 
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4. Evaluation of Relocation Manager Stakeholder Sub-Group 

Figure 16 presents the results of the evaluation for the relocation manager 
stakeholder sub-group. The FSMP satisfies the relocation managers’ criteria by a 
substantially larger margin than the other two alternatives. This outcome is 
consistent with the similarities between the relocation management concept and 
the focus of the FSMP. In almost every regard, the values relocation managers 


place on the criteria are comparable to the values I assigned to the FSMP. 


Relocation Managers | otal Difference 
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Figure 16. Relocation Managers Total Difference 


>: Evaluation of Industry Stakeholder Group 
Figure 17 presents results of the evaluation for the industry stakeholder 
group. As a whole, industry stakeholder criteria are best satisfied by the FSMP. 


The FSMP offers the greatest congruity with stakeholder expectations for 
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improving quality of life for the servicemember. There is a degree of equality 
between the SAM, MTMC and the FSMP when evaluating the ability of these 
alternatives to meet industry stakeholders desires for reducing total cost and 
maintaining a competitive market. Industry stakeholders do not appear to support 


either the SAM pilot’s approach to provider qualifications or the MTMC pilot’s 





use of DFAS as a payment option. 


6. 


Figure 18 presents the results of the evaluation for the servicemember 
stakeholder group. Overall, servicemembers demonstrate a preference for the 


FSMP over the other alternatives. Servicemembers prefer the MTMC alternative 
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Figure 17. Industry Total Difference 


Evaluation of Servicemember Stakeholder Group 
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with regards to claims settlement. However, they value the move management 


services provided by the FSMP. 
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Figure 18. Servicemembers Total Difference 


C. CHAPTER SUMMARY 

This apie: presents the data analysis necessary to support the evaluation 
of the three alternatives to reengineer the DoD’s personal property program using 
a stakeholder approach. The policy alternatives wate evaluated from each 
stakeholder perspective as well as an aggregate perspective using a single set of 
criteria. The result of this process is the identification of the Full Service Moving 
Program as the alternative that best satisfies each stakeholders group’s criteria. 


Chapter VI draws conclusions and recommendations based on this analysis. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 

The Military Traffic Management Command (MTMC) is currently 
reengineering the DoD Personal Property Shipment and Storage Program. DoD is 
conducting three pilot projects in support of this effort. Each pilot project 
represents a policy alternative for improving moving services for military families. 
This thesis evaluates the policy alternatives for reengineering the DoD personal 
property program using a stakeholder approach. 

The study provides a model for evaluating policy alternatives. Based on 
this model, evaluation criteria were established. Values were determined for these 
criteria by interviewing a convenience sample of stakeholders. Policy alternatives 
were then analyzed from each stakeholder perspective. Finally, a policy 
alternative was identified that best satisfies the criteria for each stakeholder as well 
as the aggregate of stakeholders. This chapter will summarize the study’s findings 


based on the original research questions. 


B. RESTATEMENT OF RESEARCH QUESTIONS 


Primary Research Question: 


Which policy alternative for reengineering the DoD’s personal property 
program best satisfies stakeholder criteria? 
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Subsidiary Research Questions: 


l. What are the policy alternatives for reengineering the personal 
property program? 

What are the criteria for evaluating for the policy alternatives? 
Who are the stakeholders for the reengineering effort? 


What value do the stakeholders place on the evaluation criteria? 


OY. ee. 2 


Which policy alternative fulfills stakeholder criteria? 


C. CONCLUSIONS AND RECOMMENDATIONS 

The Full Service Move Program (FSMP) best satisfies the criteria of all 
three stakeholders groups in aggregate. The extent to which the FSMP met 
stakeholders’ expectations varied from stakeholder to stakeholder. However, the 
FSMP met shah values for improving quality of life for the servicemember 
to the greatest extent. 

All stakeholder groups place the highest value on improving the 
servicemember’s quality of life. Stakeholders are consistent in placing the highest 
values on the following criteria: 


® Full Replacement Value for Personal Property 


@ Reduced Claim Preparation Requirements 

e Reduced Claim Settlement Period 

e Providers’ Qualification Based on Past Performance 
e Single Point of Contact for the Move Process 

e Improved Shipment Tracking 

@ Improved Delivery Coordination. 
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Stakeholders place the lowest value on reducing the impact on small 
business. Analysis of the interview results demonstrated that the following criteria 


are the most divisive or lowest valued: 


e Claims Settlement with the Government 

@ Inclusion of the Service Contract Act 

e Real Estate Assistance | 

@ Continued use of the Current DoD Solicitation Process 


6 Distribution of Traffic by the Government 
® Payment by DFAS 


@ Small Business Prime Contract Set-Asides. 


During the course of the interview process, stakeholders, especially 
servicemembers, consistently placed a high value on improving the quality of life 
for servicemembers. Moreover, there is a consensus among stakeholders on issues 
relating to providing full replacement insurance and improving the claims process. 
Additionally, many of the expanded services that servicemembers place the 
highest value on can be provided under the current system. Servicemember 
satisfaction can be substantially improved by implementing non-divisive quality of 
life improvements. The most divisive issues identified in this study are mainly 
related to efforts to reduce total cost and maintain a competitive market. 
Disagreement between stakeholders on aspects of these two goals has been the 
primary cause of delays and difficulties in developing and testing alternative 


polices. Specifically, government’s plan to use a relocation management company 
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to manage the personal property program has been vehemently opposed by 
transportation oui 

In expanding the Hunter Pilot into the FSMP, the government was forced to 
compromise on a number of critical components of this alternative. These 
compromises have diminished the impact a relocation management company can 
have on the personal property program. In its proposed version, the FSMP does 
not provide the relocation management company with the authority it has over 
carriers in a purely commercial environment, yet continues to hold the relocation 
company to commercial standards. Thus, the commercial move concept can not 
be adequately tested in a diluted form. A short-term solution to this impasse, may 
be an eepasion of the SAM or MTMC pilots or the development of a separate 
pilot to test the feasibility of achieving quality of life goals without the use of a 


relocation management company. 


D. LIMITATIONS 
This study faced a number of limitations that are common to policy 
analysis. These limitations can be grouped into three major categories: scope 
limitations, institutional obstacles, and technical obstacles. Each of these 
limitations provides a challenge to the evaluation process. 
1. Scope Limitations 
The scope of this thesis was limited to an evaluation of the current policy 


alternatives. No conscious effort was made to identify other alternatives or 
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solutions to the presenting problem. Interviewees did provide other options that 
did not fit within the context of this study. Furthermore, evaluation criteria were 
gauged to assess the differences between alternatives and determine the values of 
stakeholders. As a result, the criteria are not useful in identifying other criteria 
that may be highly valued by the stakeholders. 

2. Institutional Obstacles 

Institutional obstacles include the influence of political pressure on the 
policy development and evaluation, the effect of multiple competing stakeholders 
on the policy evaluation process, and the difficulties in assessing a complex public 
policy initiative. The direction the reengineering effort has taken has been 
influenced by the dynamics of the political environment. Furthermore, the 
complexity and size of the personal property program coupled with numerous 
stakeholders makes it difficult to develop criteria capable of encompassing all 
aspects of the reengineering effort. 

The policy evaluation effort faces significant obstacles unique to the public 
sector. Political maneuvering can often cause one policy alternative to be favored 
at the expense of another. Furthermore, restrictions may be placed on the 
evaluation process that limit the full range of options or solutions available to the 
decision-makers. The reengineering effort is not exempt from these political 
dynamics. DoD has tended to emphasize the FSMP at the expense of the other 


pilots, especially the SAM alternative. DoD has also concentrated on developing 
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an alternative that is palatable to all stakeholders. As a result, pilot projects have 
been used to determine which aspects of the program can be implemented 
successfully, rather than a means to evaluate three exclusive alternatives in an 
environment that encourage innovation and learning from mistakes. 
Compounding this problem is the vagueness of stakeholders in articulating 
their sub-goals and goals. A critical step in the evaluation process is developing 
adequate evaluation criteria. However, the presence of so many powerful 
stakeholder groups forced the evaluator to rite subjective judgements as to 
which goals and sub-goal to recognize and at what level in the mapping process to 
place them. Furthermore, the complexity of the personal property made it difficult 
to discern and capture the nuances embodied in certain criteria. Bonser, 
McGregor, and Oster point out that “the real world of public policy is messy, and 
it is not usually possible to assemble all of the information that one would like in 
order to prepare a complete assessment of all conceivable policy options. Most — 
inquires are under time and financial pressures and suffer from incomplete or 
inadequate data (Bonser, McGregor, and Oster, p. 5 0).” A concerted effort was 
made to validate assumptions and probe complex — associated with the 
reengineering effort throughout the evaluation and ime collection process. 


3. Technical Obstacles 


Technical obstacles presented themselves in the form of time and resource 


constraints. The use of a convenience sample provides a means to identify the 
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stakeholder values relating to the reengineering effort. However, a convenience 
sample is not arandom sample. The use of a non-random small sample introduces . 
the eeuabititiy of a bias in any results. Thus, the results cannot be generalized to a 
large population. 

Additionally, criteria are equally weighted for the purposes of this study. 
The values of the stakeholders were used to choose between the policy alternatives 
on a criterion by criterion basis. There was no attempt to weight criteria 
differently in calculating the extent each policy alternative fulfilled stakeholder 


expectations as a whole. 


E. GENERAL COMMENTS 

During the course of the interview process, various respondents often 
provided additional comments. Stakeholder comments tended to focus on issues 
that are important or of concern to them personally. Topics discussed here are 
representative of the comments received from individual stakeholders. 

A number of comments referred to changes to the damage claim process. 
One issue is the lack of true full replacement protection for the servicemember in 
any of the proposals. Each proposal iia the liability of the carrier, but this 
liability is based on a dollar per pound formula. There was also concern raised 
over the need for an arbitration mechanism when the claim is settled with a carrier. 

Another concern was that by requiring binding estimates, there is an 


incentive for transportation providers to overcharge servicemembers. Carriers are 
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compelled to inflate estimates in order to reduce the risk of underestimating the 
true cost of the additional service. Anecdotal evidence suggests that carriers do 
not lower billing charges if true cost is less than the estimate. Efforts to limit 
servicemember liability may prove more successful by managing the process, and 
not by mandating the result. 

Finally, numerous “horror stories” were told by servicemembers during the 
interviews. A central theme to these stories is the frustration servicemembers have 
with the current system. Servicemember often feel “powerless” during the move 
process. A low quality move process is often coupled with poor customer service 
and ineffective feedback to the servicemember on complaints. Servicemembers 
are unable to ensure they receive the best carrier and when they receive poor 


service they feel no one is held responsible. 


F. RECOMMENDATIONS FOR FURTHER STUDY 

This study focused on evaluating the policy alternatives for the 
reengineering of the DoD Personal Property Shipment and Storage Program from 
a stakeholder approach. This study did not seek to explore the dynamics of the 
developing public policy, nor did it wissen to address implementation issues. 
Further research would be useful in determining critical steps and possible 
obstacles in the development and implementation processes. 

The major contribution of this study is the development of a model to 


assess stakeholder views and values on alternative policies. Two additional steps 
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are needed to generate results that will sustain a scientific model. Further studies 
should attempt ss taisaanlia these two steps. The reliability of this study can be 
greatly increased by an assessment of the pilot projects by another evaluator or 

evaluators. Furthermore, the use of a random sample will allow the results of the 


analysis to be used to make inferences about the characteristics of a population. 
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APPENDIX A. DEFINITIONS 





The following terms have been defined as listed below for the purposes of this 
study: 


AMSA: The American Moving and Storage Association is a national trade 
association of the moving and storage industry. 


Best Value: A procurement strategy that takes into account both cost and quality 
in awarding contracts. 


CINC: Commander in Chief refers to the commander of a Unified Command. 


Criterion: A performance measure. 





DOD: Department of Defense. 


DFAS: The Defense Finance and Accounting Service provides responsive, 
professional finance and accounting services to the Department of Defense. 


EWG: Executive Working Group 


FSMP: The Full Service Move Program is an expansion of the Hunter Pilot with 
certain modifications based on earlier lessons learned from the Hunter Test. 


GAO: The General Accounting Office is the investigative arm of Congress. GAO 
is charged with examining matters relating to the receipt and disbursement of 
public funds. GAO performs audits and evaluations of government programs and 
activities. 


Goal: A target of an activity or policy expressed as a measurable outcome. 


HGFA: The Household Goods Forwarders Association is a national trade 
association that comprises all facets of the moving industry, including moving and 
storage firms, national van lines, line-haul firms, international and domestic 
forwarders, port agents, claims adjusters, material providers, insurance providers, 
technology firms, and others. 


Hunter Pilot: Managed by the Department of the Army as a separate pilot effort 
to determine whether the commercial business practice of outsourcing can 
alleviate known problem areas in the personal property program. 
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Industry: Includes both transportation providers and relocation managers. 
IPPT: Integrated Product Process Team. 


Military Mobility Coalition: Is an ad hoc organization formed in the spring of 
1996 to represent the relocation management industry's interests in working with 
the Department of Defense's household goods reengineering initiatives. 


MTMC: The Military Traffic Management Command is a sub-command of the 
United States Transportation Command that provides the Department of Defense 
worldwide single port management, transportation, and traffic management 

- services; deployment planning and engineering. 


MTMC Pilot: Managed by the Military Traffic Management Command in an 
effort to correct deficiencies in the personal property program through 
reengineering. 


OSD: The Office of the Secretary of Defense is the organization that establishes 
and reviews policies, and coordinates activities for the Secretary of Defense. 


Personal Property Shipment and Storage Program: The Department of 
Defense (DOD) program that is managed by the Military Traffic Management 
Command.. Provides for the shipping and storing of DOD military and civilian 
employees' household goods and personal effects. | 


Policy Analysis: A systematic approach to making policy choices. Its purpose is 
to assist policymakers in choosing a preferred course of action amid multiple 
complex alternatives. | | 


Policy Evaluation: Emphasizes examining alternative policies, as contrasted to 
describing them or explaining why they exist. 


PPSO: Personal Property Shipping Offices manage the daily operations of the 
personal property program. 


Relocation Managers: Provide a wide range of relocation services through an 
integrated network of transportation providers, real estate agents, and move 
coordinators. 
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SAM: The Servicemember Arranged Move is managed by the Department of 
Navy as a separate pilot effort to test the option of allowing servicemembers to 
choose their own carrier as well as coordinate their move with that carrier. 
SECDEF: Secretary of Defense. 


Stake: A claim that each stakeholder has on the organization or program. A stake 
may be tangible or intangible, explicit or implicit. A stake may also be political, 


economic, social or psychological in nature. 


Stakeholder: Those parties who either affect or are affected by an organization's 
actions, behavior, and policies 


Sub-goal: A set of activities that support a goal. 

Transportation Providers: Includes both motor carriers and freight forwarders. 
TMO: Traffic Management Offices are base activities that arrange the movement 
of household goods by coordinating with the commercial carrier industry and the 


military customer. 


USTRANSCOM: The United States Transportation Command ts responsible for 
planning and coordinating strategic mobility for the United States Armed Forces. 
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APPENDIX B. QUESTIONNAIRE 


Name: Stakeholder Group: G_ I Sub-Group: 


S 
Sub-goals Measurement Criteria and Sub-criteria Extent 
Valued 


| Contracting Method —s| FAR-based Cis‘ ‘és 
Loss and Damage Claim | FullReplacement  C‘(‘(‘C*S*S*S*S™S™*™S™S™SC*~drSC 
Direct Claim Settlement 


Reduced claim preparation requirements 
Reduced claim settlement period 
Financial stabilt 

Ensured capaci 

Past performance 

Inclusion of Service Contract Act 
Inconvenience payments 


Provider Performance Customer Surveys 
Measurement Random Inspections 
Periodic Performance Assessment 


Move Management Pre-move counseling 

Single point of contact move management 

Binding estimates for excess costs 

Destination information 

Shipment tracking 

Delivery coordination 

Real Estate assistance 


Current DOD solicitation process 
International 
Commercial Tariff 





Provider Qualification 





Pricing Strategy 


GSA Tender 1-W 
Single-Factor-Rate 
Commissions for move manager 


Traffic Allocation Consolidation) Fewer contracts providing more services per contract 
Traffic distribution based solely on quali 
Traffic distribution performed by: 


Move Manager 
Servicemember 
Government 


Elimination of the Traffic Distribution Record 
Reduction in Contracts awarded 
Increase length of Contract award 





EDI capability required 


Category and Service Reduction in number of movement and storage categories 
Structure 


Traffic Structure Change current structure of AOR to RA 


Payment DFAS 
| Bank Service Contract 
Government Credit Card 
Application of Small Business Act 
Small Business prime contractor set asides 
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My goal is determine the value you place on a number of criteria relating to the 
reengineering of the personal property program (moving and storage of household 
goods). I will name an objective and I would like to know how important you feel 
that objective is to any effort to change the current personal property program. 


Please rate the objectives using the following scale: 


0 = Not applicable 

1 = Not Important 

2 = Low Importance 

3 = Medium Importance 
4 = High Importance 

5 = Critical 
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APPENDIX C. INTERVIEW RESPONSES 
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(6 goals and subgoals) 
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(1 aggregate results) 
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